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Abstract 

Engineered and infrastructure systems deteriorate (e.g., loss capacity) 
as a result of adverse environmental or external conditions. Modeling de- 
terioration is essential to define optimum design strategies and inspection 
and maintenance (intervention) programs. In particular, the main pur- 
pose of maintenance is to increase the system availability by extending 
the life of the system. Most strategies for maintenance optimization fo- 
cus on defining long term strategies based on the system's condition at 
the decision time (e.g., t — 0). Ifowever, due to the large uncertainty 
in the system's performance through life, an optimal maintenance policy 
requires both permanent monitoring and a cost-efficient plan of interven- 
tions. This paper presents a model to define an optimal maintenance 
policy of systems that deteriorate as a result of shocks. Deterioration 
caused by shocks is modeled as a compound Poisson process and the op- 
timal maintenance strategy is based on an impulse control model. In the 
model the optimal time and size of interventions are executed according 
the the system state, which is obtained from permanent monitoring. 

Keywords: Impulse control, compound Poisson process, maintenance, 
optimization, shock model 

1 Introduction 



Engineered and infrastructure systems deteriorate as a result of the normal use 
or due to external demands imposed by adverse environmental conditions. The 
main challenge in modeling deterioration is to manage the damage accumula- 
tion mechanisms and the associated uncertainties. Deterioration mechanisms 
can be divided into progressive (e.g., corrosion, fatigue) and shock-based (e.g., 
earthquakes, blasts) [H]. 
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In the particular case of large infrastructure, progressive deterioration can be 
caused by, for instance, chloride ingress, which leads usually to steel corrosion, 
loss of effective cross-section of steel reinforcement in RC structures, concrete 
cracking, loss of bond and spalling [3]. The details of these deterioration mecha- 
nisms are beyond the scope of the paper but are well described by, for instance, 
[7] and [5]. On the other hand, deterioration caused by extreme events is usually 
associated with earthquakes, hurricanes or blasts (including both accidents and 
terrorists attacks). Extensive research has been carried out on mathematical 
models for shock degradation in infrastructure and in other types of engineered 
artifacts; for more details see [5], [J, [TO], [TT], 0, [H], [10] and [TO]. 

A maintenance program is as a set of actions directed to keep a deteriorating 
system (e.g., machine, building, infrastructure) operating above a pre-specified 
level of service; thus, maintenance is carried out to improve the availability or 
to extend the life of the system [TO] [TO] . The long-term benefits of an optimum 
maintenance strategy include: improvement of the system reliability, replace- 
ment cost reduction, system downtime decrease and better spares inventory 
management [TO] . 

Frequently, a comprehensive maintenance program includes preventive and/or 
corrective or reactive actions [2, [1]. Preventive maintenance involves all ac- 
tions directed to avoid failure or to avoid higher cost at a later stage by keeping 
the component in a safe or operational condition. Preventive maintenance is 
frequently carried out without knowing the actual state of the component at 
the time of the intervention. On the other hand, corrective maintenance fo- 
cuses on interventions once failure has been identified. Corrective maintenance 
is frequently more expensive than preventive maintenance since the cost may 
include, in addition to the repair cost, downtime costs or replacement of undam- 
aged system components. While preventive maintenance is commonly carried 
out at fixed time intervals, corrective maintenance is performed at unpredictable 
intervals because failure times cannot be known a priori [16] [17) . 

Most work on defining maintenance strategies focuses on establishing action 
plans of interventions [16J. However, for infrastructure systems or large sys- 
tems with long expected lifetimes (e.g., bridges that last over 75 years), these 
approaches are not realistic. Over time, the system functionality may be modi- 
fied, some unplanned demands may appear or technology changes, thus, forcing 
changes in the original maintenance plans. Predefined long-term maintenance 
plans are rarely implemented or they need to be modified as new information 
becomes available. 

Given the uncertainty associated to the component's performance, the best 
maintenance policy should be based on a permanent monitoring strategy that 
leads to optimum interventions. This paper presents a maintenance strategy 
based on impulse control models in which the time at which maintenance is 
carried out and the extent of interventions are optimized simultaneously to 
maximize the cost-benefit relationship. In the model the optimal time and size 
of interventions are executed according the the system state, which is obtained 
from permanent monitoring. 

The paper is organized as follows: the basic concepts of the impulse control 
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model are presented in section II and, in section III, some basic theorems are 
outlined. The core of the proposed optimum strategy is presented and discussed 
in section IV and a description of the numerical solution is presented in section 
V. Finally, in section VI the proposed approach is illustrated with an example 
and the main results and conclusions are summarized in section VII. 

2 Impulse Control Model 

Let's define a system component (e.g., bridge) whose performance is defined 
by a random variable R; for instance, it can be the component's reliability. 
Furthermore, assume that the component is subject to shocks, which occur 
according to a Poisson process, and every shock causes a random amount of 
damage S. Thus, if (17, J^, P) is the probability space in which we define all the 
stochastic quantities, we define the process R — {Rt,t > 0} 

Nt 

Rt=ro-Y, (1) 

i=l 

where N — {Nt,t > 0} is an homogeneous Poisson process with intensity A > 0. 
The sequence of the sizes of the shocks {S'ijigN are independent and identi- 
cally distributed random variables with probability distribution F on (0,oo). 
We assume that {^ijigN is independent of the Poisson process N. The initial 
reliability level is i?o- = ''o (Fig. [TJi). 

Definition 2.1. A maintenance policy for the system is a double sequence v = 
{(ri,Ci)}ieN of intervention times Ti at which the performance is improved an 
amount Q (in R-units). The policy is an impulse control if satisfies the following 
conditions: 

1. < Ti < Ti+i for all i e N, 

2. Ti is a stopping time with respect to the filtration Tt — <7\Rs-\s < i\ for 
t > 0, 

3. Q is a J>. -measurable random variable, 

Note that the class of impulse control policies is very general and include, 
in particular, policies with fixed time interventions. 

Given an impulse control t^, the controlled process i?" = {Rt,t > 0} is de- 
fined by 

Nt oo 

Rt^ro-Y.S. + J2QIir.<t}. (2) 

i=l i=l 

where I^nKt} is the indicator function (Fig. [TJa). 

Total failure occurs when the system performance falls bellow a pre-defined 
threshold k* with < k* . Thus, the time of total failure of the controlled 
process is denoted by 
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Figure 1: Description of the impulse control model. 
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= mi{t > 0\R'^ < fc*}, 



(3) 



and it is assumed that if the system reaches the threshold the process is stopped. 
We denote by r the time of total failure of the uncontrolled process R. With- 
out any loss of generality we assume that k* = 0. While k* denote a lower 
limit for the process, we also assume that there is a maximum (i.e., optimum) 
performance level O that cannot be improved. 

Any intervention (i.e., maintenance) at time depends on the state of the 
system just before the intervention R^-- Therefore, the set of possible actions is 
[0,0 — Rn - ] and we call I the set of impulse controls such that Q S [0,0 — R^ - ] 
for all i. We will only consider maintenance policies that are impulse controls 
and maintenance policies that are impulse controls in I. 

If we denote ^ral'] •= lE[-|i?Q_ = rg], for a given v and initial component 
state To G [0,0], the expected profit (Benefits-Costs) can be computed as: 



J(ro,!/) -E, 



JQ 



(4) 



where G is a non-negative continuous, increasing and concave function on [0, O] 
with G(0) =0. O is continuous and increasing in both variables function such 
that C > 0, and S is the discount factor. The term e"''* corresponds to the 
discounting function used to evaluate the net present value. Note that the first 
term in equation Q corresponds to the discounted benefits; where the function 
G can be interpreted as an utility function. On the other hand, the second term 
describes the discounted costs of interventions with C (r, Q the cost of bringing 
the system from level r to level r + 

The objective of the model is then to find the policy that maximizes the 
profit among all admissible impulse controls; in other words, we want to find 



F(ro) = sup J(ro,i^) (5) 

for a given level tq in the state space [0,0]. Note that it is very difficult to 
calculate ^(^o) directly from ([S]). First, given a policy ly, we can use Monte 
Carlo simulations to estimate the expected profit J{rQ,i>). Then, we have to 
repeat this process for all possible policies i', which clearly cannot be obtained 
at a reasonable computational cost. 

Instead, we will solve the problem for all r G [0, O] at the same time, that 
is, we want to find the value function 



y(r) = sup J(r,i/). (6) 

Although apparently this is a harder problem, we will characterize V as the 
unique solution of certain equation (that does not involve expectation) and solve 
this equation numerically. 
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From the definition of tfie value function, we can easy see that V > since 
we can choose to do nothing. Also, V{0) = and V is bounded. We will use 
this to characterize the function V. 



3 Preliminaries 

In this section we will present some definitions and fundamental concepts that 
are required to find V in equation ©. We start with the following lemma whose 
proof is presented in the Appendix. 

Lemma 3.1. Let T be a stopping time with respect to the filtration J-t- Then 
for all r £ [0, O] 



V{r) > Er 



/ e~^'G{Rs)ds + e-^^V{RT)I{T<r} 
Jo 



(7) 



Furthermore, we have equality in ([7]) if it is not optimal to intervene the 
system before T. 

In order to characterize the value function V in equation (|6]) we need to 
define two operators. The first one is the intervention operator Ai defined as 

Mf{r)^ sup f{r + 0~C{rX) (8) 

0<C<O-r 

for a given function / defined on [0, O] and r in the same interval. We are 
interested in applying M to the function V. Hence, if we consider any policy u 
such that Ti — and write = (0, C) U j> = (0, () U {{n, Ci)}i>2, then 

Vir) > J{r, v) = -C(r, () + J{r + C, i>), 
and since v is arbitrary we obtain 

F(r)>F(r + C)-C(r,C). 
Taking the supremum over all admissible C, we obtain that 

V{r)> sup V{r + C)-C{rX)=MV{r). (9) 

0<C<O-r 

The second operator is the infinitesimal generator of the uncontrolled process 
R, that is: 



Af{r) - A f{r - s)dF{s) - f{r) j . (10) 
The infinitesimal operator has the property that the process 

e-''f{Rt) - f{r) + e-*'^ {6f{Rs) - Af{Rs)) ds (11) 
Jo 
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is a Martingale with respect to Jt, for bounded / (see [13] and [H]). Taking 
expectation in (jlip and using Optional Sampling Theorem we obtain the so- 
called Dynkin's Formula: given Ti < T2 almost sure (a.s.) finite stopping times, 
then 



E[e-^^V(^Tj-e-^^V(i?Tj] 

= E I \-''{Af[Rs)-5f{R,))ds 



(12) 



Bellow, we will use this Formula with / replaced by V . 

4 Optimal maintenance policy 

Since the process R is Markovian, the future is independent of the past given the 
present. Thus, in order to obtain an optimal policy it is necessary to differentiate 
between the component states at which an intervention is required and those 
where there is no need to intervene the system. It is important to stress that 
because of the Markovian property, this classification will always be the same 
and will only depend on the state of the system. 

We use now the intervention operator M to describe the optimal policy. 
Using equation ([9]) F > MV , we can divide the state space [0,0] into the 
subsets 

A^{re [0, O] : V{r) = MV{r)} 

and 

B = {re [0, O] : V{r) > MV{r)}. 

For r G A we must intervene the system immediately and improve the per- 
formance process by C*, where 



MV{r) = V{r + C)-C{r,C) 

sup F(r + C)-C(r,C). (13) 

0<C<O-r 

Therefore, we call the set A the intervention region. 

Now, for r e _B we must do nothing and let the system evolve. Therefore, 
we obtain equality in ([7]), and using Dynkin's Formula we have that dV{r) — 
AV{r) = G{r). We call the set B the no intervention region (Fig. (2). 

The previous discussion is the intuition behind the following theorem (proved 
in Appendix). 

Theorem 4.1. The value function V solves the equation 

min{5V{r) - AV{r) - G{r), V{r) - 7WV^(r)} = 0, (14) 
/or all r £ [0, O] . 
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Figure 2: Description of the impulse control model. 



Remark It is possible that C* is not attainable in the equation above. In 
this case there is not attainable optimum policy, but we can find controls with 
expected profit within any degree of accuracy from the value function V . 

To obtain a full characterization of the value function V (and the optimal 
policy), we show now that V is the only solution of ([T3l) . 



Theorem 4.2. Let g he a non-negative hounded function on [0, O] that solves 
(fT3l) such that g(0) = 0. Then g^V. 



Proof. Let v = {(Ti,Ci)}ieN £ T. and initial reliability level r 6 [0,0]. Using 
Dynkin's Formula, for t > we have 



+ E^, 



E 



-St, 



e-'^{~5g{R'^,)+Ag{K))ds 
{g{K^_+Q-9{K-)) 



Since g solves ([T^. then 

E.[e-^(*^^'')5(i?Mr.O] 



<g[r) - E, 



-5s 



G{R'',)ds 



Er 



E 
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Letting t oo, bounded convergence we get J(r, v) < gir) and taking sup over 
I we obtain 

V{r) < gir). 

To prove the reverse inequality, let e > and define the following admissible 
impulse control v^: 



and such that 



Note that 5g[R^J') — AgiWf) = GiRf) for s between interventions. Therefore, 
from the previous calculations we obtain 



>g{r) - E,, 



/ e-'^'G{Rf)ds 
Jo 



J2 ^'^^CiR^-.Q) 



<tf\T" 



Letting t oo again, we get gir) < J(t^^, r) + e < V{r) + e. Since e is arbitrary 
we get the reverse inequality 

gir) < V{r). 



□ 



5 Numerical solution 

To obtain the optimal policy is necessary to find the value function V by solv- 
ing equation (jl3p . Once we have V we can compute the intervention and no 
intervention regions. Thus, for r £ B, we have that 

SV{r)~AV{r) - G{r) = 0. 

Using the definition of the infinitesimal operator A (equation pop ) with 
f — V and solving the above equation for V{r) it is obtained that 

V(.r) = ^ |G(r) + A V{r - . (15) 

On the other hand, for the region where interventions are required; i.e., 
r G A, we have that 

V{r) - MV{r) = 0. 
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Then, usmg the definition of the intervention operator (equation ([5])), V{r) 
satisfies: 



Vir)= sup Vir + 0-Cir,C). (16) 

0<C<O-r 

To approximate V we foUow the Jacobi iteration method described in [5]. 
First, we discretize the interval [0, O] in iV = ^ intervals and initizialize the 
vector = G Now, given Vj^ we compute 

max I ^ I Gijh) + ^ E ^» [j - I ' -^^n [j] I , 

where pi is the discretized density of F for ih and 

M [j] = max . V^i [j + z] - C{jh,ih). 

0<i<N-\-l—j 

Note that Vj^[0] = for all n and all h since F(0) = 0. 
We continue iterating over n until 
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for a given error tolerance e 



6 Illustrative example 

Consider an infrastructure system subject to shocks (e.g., earthquakes) whose 
occurrence times follow an exponential distribution with rate A = 0.5. The 
sizes of the shocks are log- normally distributed with n = 0.3 and a ~ I; and, 
consequently, distribution parameters fiiog = 2.23 and aiog = 2.9. Furthermore, 
assume that the performance (state) of the system is permanently monitored. 
This means that it is possible to know the state of the system when required (e.g., 
at periodic inspection times). The system state (which should be in practice 
measured in physical units) is normalized and evaluated within the interval [0,1]; 
where 1 means that the system is in "as good as new" condition, and k* ~ 
indicates that it is not in operating condition. The objective of the study is to 
define an optimal maintenance policy. 

For the purpose of this example, the following assumptions are made. The 
function G{r) (equation (|31)), which is the utility function is given by: 

G(r) = C-(l -e-"'-), (17) 
a 

where C — 5 and a — 2. Note that this curve has the form of an exponential 
risk aversion utility function. On the other hand, it is assumed that the costs 
associated to an intervention are given by the following function (equation (|3])), 
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C{rX)=r + C + K 



(18) 



where the constant K = 0.1 reflects the fixed costs of any intervention. Note 
that the intervention costs are proportional to the current state of the system 
and grow with the square of the size of the intervention. For both utility and 
cost, these values are discounted to the time of the decision by using a discount 
factor 6 = 0.2. 

The outcome of the approach consists of two parts. First, it is necessary to 
define, for every system state r, the intervention intensity ^ that maximizes the 
expected profit (equation 2]). This requires dividing the system in two states: a 
region (state space values) where no intervention is required and a set of values 
for which it is necessary. Thus, for a given system state obtained (measured) 
at the time of inspection, the intervention level that maximizes the expected 
profit at that particular time (i.e., discounted) can be obtained from this result. 
The second result, is the value function V that provides the optimum expected 
profit value obtained if the required actions (defined previously) are conducted. 
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Following the numerical approach presented in section[5]we obtain the results 
after 8 iterations at a minimum computational time. The division between the 
intervention and not intervention states (i.e. regions A and B) can be observed 
in Fig. [21 Clearly, if the system is operating at a level r > 0.328 there is 
no need for an intervention. However, if an inspection indicates that its state 
is r < 0.328 and intervention is required and the size of the intervention is 
shown in the figure. For instance, if after an inspection at time t the system 
state is Rt — 0.6 no intervention is required, but if Rt = 0.3 and intervention 
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of magnitude C = 0.64 will be necessary to maximize the profit. This means 
that immediately after the intervention the system will be at state 0.94 (i.e., 
0.3+0.64=0.94). Notice that Fig. [3] does not change with time as mentioned 
earlier. If in the next inspection the observed state of the system is, say 0.6, no 
intervention will be necessary 

Furthermore, if this maintenance pohcy is carried out, the maximum ex- 
pected profit can be observed in Fig. |4l Therefore, if after an inspection at 
time t the system state is Rt = 0.3, the maximum expected profit that we 
can obtain, by following the above policy, is ^(0.3) = 3.77. Otherwise, if the 
inspection yields a state Rt = 0.6, we can obtain a maximum of V^(0.6) = 4.13. 

5, ^ ^ ^ . 1 



4.8 




2 g I I I I I I 

■ 0.2 0.4 0.6 0.8 1 

System state (e.g., strength/capacity) 



Figure 4: Maximum expected profit. 



7 Summary and conclusions 

The paper presents an approach to define the optimum maintenance policy of 
a system that deteriorates with time as a result of shocks. The deterioration 
process is modeled as a compound Poisson process. The proposed maintenance 
strategy follows an impulse control model that requires the permanent (or at 
least frequent) monitoring of the system state. Then, at every inspection time 
the model can be used to make a decision as to whether the system should be 
intervened or not. In case of requiring an intervention, the extent of the opti- 
mum repair can be obtained from the model. The decisions based on the model 
guarantee that the net present value of the utility at the time of the interven- 
tion is maximum. It is suggested in the paper that this maintenance approach 
is of particular importance for engineering systems that operate under adverse 
environments for long time periods (e.g., > 25 years); for instance, physical in- 
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frastructure (e.g., bridges, highways). Traditional maintenance strategies define 
long term inspection and maintenance plans at a given point in time (usually 
t = 0) without considering possible variations in the system's use or condition, 
and the technology available at the time of the decisions. Thus, it is argued 
that the best maintenance policy in these cases can be obtained by combining 
both permanent monitoring and optimum interventions. 

Lemma 13. 1[ Let T be a stopping time with respect to the filtration Tt ■ Then 
for all r G [0, O] 



V{r) > Er 



/ e~^'G{Rs)ds + e-^^V{RT)I{T<r} 
Jo 



(7) 



Furthermore, we have equality in ([7]) if it is not optimal to intervene the 
system before T . 

Proof. Let r G [0, O] and v — {(r^, Ci)}ieN £ I, then J(r, v) = E,. [rj] where 



V = 



Jo ^.^^^ 



Given T a stopping time with respect to the filtration J-t , the control v + T = 
{{Ti + T,Ci)}i€N is also an admissible control. If we call 8t the usual shift 
operator (see [13J), from the strong Markov property of the process R we have 



J(r, ly + T) 



Er 



TAt 



-5s 



G{Rs)ds + e~^^eTVl{T<r} 



TAt 



e~''G{Rs)ds + e^^^En^ W{t<t} 



= Er 



TAt 



e-^'GiRs)ds + e-^'^JiRr, j^)I{t<t} 



Taking sup over v & I we obtain If it is optimal not to intervene before T, 
then V{r) = sup^gj J{r,v + T). □ 

Theorem 14.11 The value function V solves the equation 

min{5V{r) - AV{r) - G{r), V{r) - MV{r)} = 0, (13) 

for all r £ [0, O] . 

Proof. Let r G [0, O]. From equation (HJ 1^ > AiV. On the other hand, for any 
stopping time T < t a.s., by Dynkin's Formula and boundedness of V, we have 

V{r) -Er[e-'^ V{Rt)] 



-5s 



{SV{Rs)-AV{Rs))ds 
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Using ([7]) we get that 



e-^' {SV{Rs) - AV{Rs) - G{Rs)) ds 



> 0. 



If we choose T = Ti, the time of the first shock, then Kg — r ior < s < Ti 
and therefore SV{r) - AV{r) - G{r) > 0. Now, suppose that V{r) > MV{r), 
hence it is not optimal to intervene at time 0, so it is not optimal to intervene 
before Ti. Then, by lemmaOwe get SV{r) - AV{r) - G{r) =0. □ 



References 

[1] T. J. Aven and U. Jansen. Stochastic Models in Reliability. AppUcations 
of Mathematics: stochastic modeling and appUed probabiUty series 41. 
Springer, New York, 1999. 

[2] R. E. Barlow and F. Poschan. Mathematical theory of Reliability. Wiley, 
New York, 1965. 

[3] E. Bastidas, P. Bressolette, A. Chateneuf, and M. Sanchez-Silva. Prob- 
abilistic lifetime assessment of rc structures subject to corrosion-fatigue 
deterioration. Structural Safety^ 31:84-96, 2009. 

[4] R. Dekker. Applications of maintenance optimization models: a review and 
analysis. Reliability engineering and systems safety, 51:229240, 1996. 

[5] R. M. Feldman. Optimal replacement for systems governed by markov 
additive shock processes. Annals of Probability, 5:413-429, 1977. 

[6] D. M. Frangopol, M.-J. Kallen, and M. Noortwijk. Probabilistic models 
for life-cycle performance of deteriorating structures: review and future 
directions. Steel construction Prog. Struct. Engng Materials, 6:197-212, 
2004. 

[7] G.-A. Klutke and Y. Yang. The availability of inspected systems subject 
to shocks and graceful deterioration. IEEE Transactions on Reliability, 
51(3):371-374, 2002. 

[8] H. Kushner and P. Dupuis. Numerical methods for stochastic control prob- 
lems in continuous time. Springer- Verlag, New York, 1992. 

[9] K. B. Misra. Reliability analysis and prediction: a methodology oriented 
treatment. Elsevier, Amsterdam, 1992. 

[10] T. Nakagawa. On a replacement problem of cumulative damage model: 
Part 1. Operational Research, 27(4):895-900, 1976. 

[11] T. Nakagawa. Continuous and discrete age replacement policies. Opera- 
tional Research, 36(2):147-154, 1985. 



14 



[12] T. Nakagawa. Maintenance Theory of Reliability. Springer Series on Reli- 
ability Engineering, London, 2005. 

[13] L. C. G. Rogers and D. Williams. Diffusions, Markov processes, and mar- 
tingales. Vol. 1. Cambridge Mathematical Library. Cambridge University 
Press, 2000. 

[14] M. Sanchez-Silva, G.-A. Klutke, and D. Rosowsky. Life-cycle performance 
of structures subject to multiple deterioration mechanisms. Structural 
Safety, 33(3):206217, 2011. 

[15] S. Thonhauser and H. Albrecher. Opimal dividend strategies for a com- 
pound poisson process under transaction costs and power utility. Stochastic 
Models, 27:120-140, 2011. 

[16] C. Valdcz-Flores and R. M. Fclclman. A survey of preventive maintenance 
models for stochastically deteriorating single unit systems. Naval Research 
Logistics Quarterly, 36:419-446, 1989. 

[17] H. Wang and H. Pham. Reliability and optimal maintenance. Springer, 
London, 2006. 

[18] Y. Wang and H. Pham. Modeling the dependent competing risks with mul- 
tiple degradation processes and random shock using time-varying copulas. 
IEEE Transactions on Reliability, 61(1):13 22, 2012. 

[19] M. A. Wortman, G.-A. Klutke, and H. Ayhan. A maintenance strategy 
for systems subjected to deterioration governed by random shocks. IEEE 
Transactions on Reliability, 43(3):439-445, 1994. 

[20] Z. S. Ye, L. C. Tang, and H. Y. Xu. A distribution-based systems reliability 
model under extreme shocks and natural degradation. IEEE Transactions 
on Reliability, 60(1) :246 -256, 2011. 



15 



